{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Character-level Language Modeling with RNN\n",
    "\n",
    "Adapted from [Practical Pytorch](https://github.com/spro/practical-pytorch/tree/master/char-rnn-generation) and [Character_Level_Language_Model](https://github.com/m2dsupsdlclass/lectures-labs/blob/master/labs/06_deep_nlp/Character_Level_Language_Model.ipynb)\n",
    "\n",
    "Steps:\n",
    "\n",
    "- Download a small text corpus and preprocess it.\n",
    "- Extract a character vocabulary and use it to vectorize the text.\n",
    "- Train an LSTM-based character level language model.\n",
    "- Use the trained model to sample random text with varying entropy levels.\n",
    "- Implement a beam-search deterministic decoder.\n",
    "\n",
    "See also Karpathy's[blog](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) and J. Johnson implementation: [torch-rnn](https://github.com/jcjohnson/torch-rnn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading some text data\n",
    "\n",
    "Sources:\n",
    "\n",
    "https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\n",
    "\n",
    "https://s3.amazonaws.com/text-datasets/nietzsche.txt\n",
    "\n",
    "http://www.theatre-classique.fr/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Corpus length: 1115394 characters\n"
     ]
    }
   ],
   "source": [
    "import unidecode\n",
    "import string\n",
    "import random\n",
    "import re\n",
    "import pathlib\n",
    "\n",
    "#to be modified!\n",
    "data_root = pathlib.Path('/home/public/')\n",
    "\n",
    "# if you prefer Shakespeare (bigger)\n",
    "text = unidecode.unidecode(open(data_root/'text/shakespeare.txt').read().lower())\n",
    "# if you prefer Nietzsche\n",
    "#with open(data_root/'text/nietzsche.txt', 'r', encoding=\"utf-8\") as f:\n",
    "#    text = f.read().lower()\n",
    "\n",
    "#with open(data_root/'text/fleurs_du_mal.txt', 'r', encoding=\"utf-8\") as f:\n",
    "#    text = f.read().lower()\n",
    "\n",
    "print('Corpus length: %d characters' % len(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first citizen:\n",
      "before we proceed any further, hear me speak.\n",
      "\n",
      "all:\n",
      "speak, speak.\n",
      "\n",
      "first citizen:\n",
      "you are all resolved rather to die than to famish?\n",
      "\n",
      "all:\n",
      "resolved. resolved.\n",
      "\n",
      "first citizen:\n",
      "first, you know caius marcius is chief enemy to the people.\n",
      "\n",
      "all:\n",
      "we know't, we know't.\n",
      "\n",
      "first citizen:\n",
      "let us kill him, and we'll have corn at our own price.\n",
      "is't a verdict?\n",
      "\n",
      "all:\n",
      "no more talking on't; let it be done: away, away!\n",
      "\n",
      "second citizen:\n",
      "one word, good citizens.\n",
      "\n",
      "first citizen:\n",
      "we are accounted poor citizens, the patricians good.\n",
      "what authority surfeits on would relieve us: if they\n",
      "would yield us  ...\n"
     ]
    }
   ],
   "source": [
    "print(text[:600], \"...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = text.replace(\"\\n\", \" \")\n",
    "text = re.sub(\"\\d+\", \"\", text) # remove numbers\n",
    "split = int(0.9 * len(text))\n",
    "train_text = text[:split]\n",
    "test_text = text[split:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a Vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total chars: 37\n"
     ]
    }
   ],
   "source": [
    "chars = sorted(list(set(text)))\n",
    "n_characters = len(chars)\n",
    "print('total chars:', n_characters)\n",
    "char_to_indices = dict((c, i) for i, c in enumerate(chars))\n",
    "indices_to_char = dict((i, c) for i, c in enumerate(chars))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(' ', 0),\n",
       " ('!', 1),\n",
       " ('$', 2),\n",
       " ('&', 3),\n",
       " (\"'\", 4),\n",
       " (',', 5),\n",
       " ('-', 6),\n",
       " ('.', 7),\n",
       " (':', 8),\n",
       " (';', 9),\n",
       " ('?', 10),\n",
       " ('a', 11),\n",
       " ('b', 12),\n",
       " ('c', 13),\n",
       " ('d', 14)]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(char_to_indices.items())[:15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "';'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indices_to_char[9]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0sAAADFCAYAAACFFmlDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAF1ZJREFUeJzt3XuwZWV95vHvM6DGiMqtJQqMrbFLg2Qg0iIZtQohYgNxIBMkqAmtAXpQKM3FVDCawgGtYExiaRUhIdIFxBtG49Azgkg1GFQk0txBRDraDI0ILSCglBfgN3/st8dNs/rCXutwdp/z/VSdOmu/e+3ffnf3vj3rfdd7UlVIkiRJkh7rP812ByRJkiRpGhmWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOmw72x0Y2s4771wLFy6c7W5IkiRJmlJXXXXVD6pqweb2m3NhaeHChaxatWq2uyFJkiRpSiW5bUv2cxqeJEmSJHUwLEmSJElSB8OSJEmSJHUwLEmSJElSB8OSJEmSJHWYc6vhTZOFJ32h1+3XnHboQD2RJEmS9EQ5siRJkiRJHQxLkiRJktTBsCRJkiRJHQxLkiRJktRhs2Epye5JLk3yzSQ3JXlna98xycVJbm2/d2jtSfLRJKuTXJ/kZWO1lrb9b02ydKx9nyQ3tNt8NEk2dR+SJEmSNNO2ZGTpYeBPq2oPYD/ghCR7ACcBK6tqEbCyXQY4GFjUfpYBZ8Ao+AAnA68A9gVOHgs/ZwDHjd1uSWvf2H1IkiRJ0ozabFiqqjur6uq2/SBwM7ArcBhwTtvtHODwtn0YcG6NXAFsn+S5wOuAi6vq3qq6D7gYWNKue1ZVXVFVBZy7Qa2u+5AkSZKkGfWEzllKshD4DeDfgV2q6s521feBXdr2rsDtYzdb29o21b62o51N3MeG/VqWZFWSVevWrXsiD0mSJEmSOm1xWEqyHfA54I+q6oHx69qIUA3ct8fY1H1U1ZlVtbiqFi9YsGAmuyFJkiRpntiisJTkKYyC0ieq6l9b811tCh3t992t/Q5g97Gb79baNtW+W0f7pu5DkiRJkmbUlqyGF+As4Oaq+ruxq1YA61e0WwqcP9Z+dFsVbz/g/jaV7iLgoCQ7tIUdDgIuatc9kGS/dl9Hb1Cr6z4kSZIkaUZtuwX7vBL4A+CGJNe2tr8ATgM+k+QY4DbgyHbdBcAhwGrgIeCtAFV1b5JTgSvbfqdU1b1t++3A2cDTgQvbD5u4D0mSJEmaUZsNS1X1VSAbufrAjv0LOGEjtZYDyzvaVwF7drTf03UfkiRJkjTTntBqeJIkSZI0XxiWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOhiWJEmSJKmDYUmSJEmSOhiWJEmSJKnDZsNSkuVJ7k5y41jb+5LckeTa9nPI2HXvTrI6yS1JXjfWvqS1rU5y0lj7C5L8e2s/L8lTW/vT2uXV7fqFQz1oSZIkSdqcLRlZOhtY0tH+4arau/1cAJBkD+Ao4KXtNn+fZJsk2wCnAwcDewBvbPsCfLDVehFwH3BMaz8GuK+1f7jtJ0mSJElPis2Gpaq6DLh3C+sdBny6qn5aVd8FVgP7tp/VVfWdqvoZ8GngsCQBDgA+225/DnD4WK1z2vZngQPb/pIkSZI04/qcs3RikuvbNL0dWtuuwO1j+6xtbRtr3wn4YVU9vEH7Y2q16+9v+z9OkmVJViVZtW7duh4PSZIkSZJGJg1LZwC/CuwN3An87WA9mkBVnVlVi6tq8YIFC2azK5IkSZLmiInCUlXdVVWPVNWjwD8xmmYHcAew+9iuu7W2jbXfA2yfZNsN2h9Tq13/7La/JEmSJM24icJSkueOXfwdYP1KeSuAo9pKdi8AFgHfAK4EFrWV757KaBGIFVVVwKXAEe32S4Hzx2otbdtHAJe0/SVJkiRpxm27uR2SfArYH9g5yVrgZGD/JHsDBawB/gdAVd2U5DPAN4GHgROq6pFW50TgImAbYHlV3dTu4s+BTyd5P3ANcFZrPwv45ySrGS0wcVTvRytJkiRJW2izYamq3tjRfFZH2/r9PwB8oKP9AuCCjvbv8ItpfOPtPwHesLn+SZIkSdJM6LManiRJkiTNWYYlSZIkSepgWJIkSZKkDoYlSZIkSepgWJIkSZKkDoYlSZIkSepgWJIkSZKkDoYlSZIkSepgWJIkSZKkDoYlSZIkSepgWJIkSZKkDoYlSZIkSepgWJIkSZKkDoYlSZIkSepgWJIkSZKkDoYlSZIkSepgWJIkSZKkDoYlSZIkSepgWJIkSZKkDoYlSZIkSepgWJIkSZKkDoYlSZIkSepgWJIkSZKkDoYlSZIkSepgWJIkSZKkDoYlSZIkSepgWJIkSZKkDoYlSZIkSeqw2bCUZHmSu5PcONa2Y5KLk9zafu/Q2pPko0lWJ7k+ycvGbrO07X9rkqVj7fskuaHd5qNJsqn7kCRJkqQnw5aMLJ0NLNmg7SRgZVUtAla2ywAHA4vazzLgDBgFH+Bk4BXAvsDJY+HnDOC4sdst2cx9SJIkSdKM23ZzO1TVZUkWbtB8GLB/2z4H+DLw56393Koq4Iok2yd5btv34qq6FyDJxcCSJF8GnlVVV7T2c4HDgQs3cR/z1sKTvjDxbdecduiAPZEkSZLmvknPWdqlqu5s298HdmnbuwK3j+23trVtqn1tR/um7uNxkixLsirJqnXr1k3wcCRJkiTpsXov8NBGkWqAvkx8H1V1ZlUtrqrFCxYsmMmuSJIkSZonJg1Ld7XpdbTfd7f2O4Ddx/bbrbVtqn23jvZN3YckSZIkzbhJw9IKYP2KdkuB88faj26r4u0H3N+m0l0EHJRkh7aww0HARe26B5Ls11bBO3qDWl33IUmSJEkzbrMLPCT5FKOFFnZOspbRqnanAZ9JcgxwG3Bk2/0C4BBgNfAQ8FaAqro3yanAlW2/U9Yv9gC8ndGKe09ntLDDha19Y/chSZIkSTNuS1bDe+NGrjqwY98CTthIneXA8o72VcCeHe33dN2HJEmSJD0Zei/wIEmSJElzkWFJkiRJkjoYliRJkiSpg2FJkiRJkjoYliRJkiSpg2FJkiRJkjoYliRJkiSpg2FJkiRJkjoYliRJkiSpg2FJkiRJkjoYliRJkiSpg2FJkiRJkjoYliRJkiSpw7az3QHNjoUnfWHi26457dABeyJJkiRNJ0eWJEmSJKmDI0vqrc8oFThSJUmSpOnkyJIkSZIkdTAsSZIkSVIHw5IkSZIkdTAsSZIkSVIHw5IkSZIkdXA1PE0d/waUJEmSpoEjS5IkSZLUwbAkSZIkSR0MS5IkSZLUwXOWNKd5/pMkSZIm5ciSJEmSJHUwLEmSJElSh17T8JKsAR4EHgEerqrFSXYEzgMWAmuAI6vqviQBPgIcAjwEvKWqrm51lgLvbWXfX1XntPZ9gLOBpwMXAO+squrTZ2lSfab0weOn9Q05RXBaanXVkyRJ2loNMbL0mqrau6oWt8snASurahGwsl0GOBhY1H6WAWcAtHB1MvAKYF/g5CQ7tNucARw3drslA/RXkiRJkjZrJqbhHQac07bPAQ4faz+3Rq4Atk/yXOB1wMVVdW9V3QdcDCxp1z2rqq5oo0nnjtWSJEmSpBnVNywV8KUkVyVZ1tp2qao72/b3gV3a9q7A7WO3XdvaNtW+tqP9cZIsS7Iqyap169b1eTySJEmSBPRfOvxVVXVHkucAFyf51viVVVVJZvwco6o6EzgTYPHixZ7TJEmSJKm3XiNLVXVH+3038HlG5xzd1abQ0X7f3Xa/A9h97Oa7tbZNte/W0S5JkiRJM27isJTkGUmeuX4bOAi4EVgBLG27LQXOb9srgKMzsh9wf5uudxFwUJId2sIOBwEXteseSLJfW0nv6LFakiRJkjSj+kzD2wX4/CjHsC3wyar6YpIrgc8kOQa4DTiy7X8Bo2XDVzNaOvytAFV1b5JTgSvbfqdU1b1t++38YunwC9uPJEmSJM24icNSVX0H2Kuj/R7gwI72Ak7YSK3lwPKO9lXAnpP2UZIkSZImNRNLh0uSJEnSVs+wJEmSJEkdDEuSJEmS1KHv31mSpI1aeNIXet1+zWmHDtQTSZKkJ86RJUmSJEnqYFiSJEmSpA5Ow5O01egzrc8pfZIk6YlyZEmSJEmSOhiWJEmSJKmD0/AkzUtO6ZMkSZvjyJIkSZIkdTAsSZIkSVIHp+FJUk/+8V1JkuYmR5YkSZIkqYMjS5I0ZVx8QpKk6WBYkqQ5zOAlSdLkDEuSpC0y9LlZBjlJ0rTznCVJkiRJ6uDIkiRpq+colSRpJhiWJEka41LwkqT1DEuSJM0Qg5ckbd08Z0mSJEmSOjiyJEnSVsJzsyTpyWVYkiRpHjJ4SdLmGZYkSVIv/g0uSXOVYUmSJM1ZBi9JfRiWJEmStoCrG0rzj6vhSZIkSVKHqR9ZSrIE+AiwDfCxqjptlrskSZLUm1MEpek31WEpyTbA6cBrgbXAlUlWVNU3Z7dnkiRJ02PI4DXkdEOnLmprN9VhCdgXWF1V3wFI8mngMMCwJEmSNM9MSyicyYDZt54Bc1ipqtnuw0YlOQJYUlXHtst/ALyiqk7cYL9lwLJ28cXALU9qRye3M/CDKaw1dL1prTV0vWmtNXS9+VBr6HrTWmvoetNaa+h686HW0PWmtdbQ9aa11tD15kOtoetNa62h6w3dt5n0/KpasLmdpn1kaYtU1ZnAmbPdjycqyaqqWjxttYauN621hq43rbWGrjcfag1db1prDV1vWmsNXW8+1Bq63rTWGrretNYaut58qDV0vWmtNXS9ofs2DaZ9Nbw7gN3HLu/W2iRJkiRpRk17WLoSWJTkBUmeChwFrJjlPkmSJEmaB6Z6Gl5VPZzkROAiRkuHL6+qm2a5W0Macurg0NMQp7VvPs7Zrzcfag1db1prDV1vWmsNXW8+1Bq63rTWGrretNYaut58qDV0vWmtNXS9re60mM2Z6gUeJEmSJGm2TPs0PEmSJEmaFYYlSZIkSepgWNJjJNk+ydtnux9PhiSXD1RnYZIbh6g13yT50QzUfF+Sdw1dV1uvJO9IcnOST8x2X9abL+8b8+VxTrMkL0lyeZIbkvxbkp171pvq99ihPtvnuiQ7J7k0yfVJvpFku4HqLkzyliFqTQvDkja0PTAvwlJV/dfZ7sOTKSO+5jUfvR14bVW9ebY7orkryZrZ7sMm/H5V/TpwOXD8bHdmJs23z/Ye3gZcVlX/BTgc+FnfgkneBlwInJrky0l+pW/NaeAXpzkgye+3owLXJvnHJNv0KHca8Kut1ocG6NufJLmx/fxRz1r/K8lVSW5KsmyAvg05qrFNkn9qfftSkqdP2KdnJPlCkuvav9nv9elUO8JzS5JzgRt57N8tm6TWzUM8zqEleU+Sbyf5KvDiHnUecxQ8ybuSvG/COt9Kcnbr1yeS/FaSryW5Ncm+E9Q8Zfw1lOQDSd45QZ0/S/KOtv3hJJe07QOmZeQlydHtaOd1Sf65Z61/AF4IXJjkj3vW+sv2evpqkk8NcHR92/bcuDnJZ5P88oT9Wv98611rJiV5YZJrkrx8wtufluSEsctTPcLRV5ILkjyvb52q+lZVfaddfBrwk741p9kQn+3jnyl9XutJXt7ey36pfb7flGTPHv06vn0/uzbJd5NcOmktRuFoN4Cq+l5V9QpLSZ4J/E/gzcBfAm8Bftyn5rQwLG3lkvwa8HvAK6tqb+ARRk/USZ0E/EdV7V1Vf9azb/sAbwVeAewHHJfkN3qU/MOq2gdYDLwjyU59+jewRcDpVfVS4IfA705YZwnwvaraq6r2BL44UN/+vqpeWlW3DVBriMc5mPY8OwrYGzgEmOiL2Ax4EfC3wEvaz5uAVwHvAv5ignrLgaMB2gjhUcDHJ6jzFeDVbXsxsF2Sp7S2yyaoN6gkLwXeCxxQVXsBTzgQjquq44HvAa+pqg/36NfLGT3f9wIOZvRv19eLGb02fw14gH6j+kPWGuyL+li9FwOfA95SVVdOWOY84Mixy0e2tmmxbshiVXVIVX1vqHpJXsfoufuxoWrORUN+prTn+grg/cBfAx+vqomnpVbVP7Tvei8H1gJ/N2kt4D+A/55kqJHGR4ECdgSoqjVV9eBAtWeVYWnrdyCwD3Blkmvb5RfObpf+v1cBn6+qH1fVj4B/5Rdf0ibxjiTXAVcwGiFZNEAfh/Ldqrq2bV8FLJywzg3Aa5N8MMmrq+r+Afp2W1VdMUAdGO5xDunVjJ5nD1XVA0zPH67+blXdUFWPAjcBK2v0txpuYIJ/t6paA9zTDjgcBFxTVfdM0K+rgH2SPAv4KfB1Rl/8X80oSM22A4B/qaofAFTVvbPcn/VeCZxfVT9pXwD+9wA1b6+qr7XtjzN6z5yGWkN/UV8AnA+8uaqu69Gna4DnJHlekr2A+6rq9oH62FtVTcuBmsdpB1jOAv5bVf1wtvsz5Yb+TDkFeC2j99m/7tu55iPAJVU10ftQkl2BdzM6qHdskt9t7dcnefYkNavqx8BxwF8xmob3N9M4wj2Jqf6jtNoiAc6pqnfPdkdmUpL9gd8CfrOqHkryZeCXZrVTj/XTse1HgImmp1XVt5O8jNHRrPcnWVlVp/Ts25DD4IM8zin2MI89iNTnOTb+b/Xo2OVHmfy992OMpjb8CqORpiesqn6e5LutzuXA9cBrGH1o3jxhvzSZDf/QYZ8/fDhkraHdD/xfRgHumz1r/QtwBKPXwDSNKk275wH3V9WtfQtV1fv6d2ekTas8rl0cdCRtiuwEbAc8hdFnSq/P5IwWT3g+cGKPMq8Ebqiqe5IcCqxMsguwps9B2qpakeR64PWMwuGfAqf26OdUcGRp67cSOCLJcwCS7Jjk+T3qPQg8c5CejY5SH57kl5M8A/gdJj9y/WxGRxEfSvISRtP65pw27eWhqvo48CHgZbPcpa3BZYyeZ09vc6Zf36PWXYyOXO+U5GnAbw/Sw+F8ntFUzZcDF/Wo8xVG0wEva9vHMxqp6vUFO8nKdsSyj0uAN6yfZptkx571hvI14PXt3IPtGOa58Z+T/GbbfhPw1SmpNbSfMXr/PzrJm3rWOo/RFKkjGAWnOWug19N69zH64jpVqur0Nu1/7ykKSkN+pgD8I6NzeD4BfLBPoTZF8F2MFux4tEep64HXJHleVd0F/DFwOvDJHn3bbuz754OMDr4N9X1yVjmytJWrqm8meS/wpTbM/nPgBGCic1PaUYavZXSS+4V9zluqqquTnA18ozV9rE2jmMQXgeOT3Azcwmgq3lz068CHkjzK6P/ybbPcn6nXnmfnAdcBdwOTng+xftTlFEbP2TuAbw3Ty2FU1c/aCb0/rKpHepT6CvAe4OtV9eMkP6HnFLz2/vMioNe0uaq6KckHgH9L8ghwDaNRsFlVVVcmWcHoS8ZdjKZT9p0mewtwQpLljEZczpiSWiS5ADh2qC+w7Xn228DFSX5UVRNNbWrPj2cCd1TVnX37NfTjHMpQr6cxzwaOZYDzYNs5Lg9V1bm9ezUzeh30GfIzJcnRwM+r6pMZLb51eZIDquqSCUueyOicoEuTAKyqqmOfaJGq+laS9wAXJfk5o/e0o4DTklxdVd+eoG9PYRQMdwJ2ZjSa3PfgyFRIzwOJkqQnSfsCdTXwhiGm0wyprfD0h1X1J7Pdl5mSZLuq+lGbh38ZsKyqrp6Cfi0E/k9bFEZzwHx4Pc2ENiJ9dVX1mWGzYc33AT+qqr8ZquZc1t6P9q+qs2e3J8NxGp4kbQWS7AGsZrRQxFQFJYCqunEefLE7sy2kczXwuWkISpqb5snraVBtGvvXAUPN7PohcO1m99qKOLIkSZIkSR0cWZIkSZKkDoYlSZIkSepgWJIkSZKkDoYlSZIkSepgWJIkSZKkDv8PpmCaOBMPzOgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1008x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "counter = Counter(text)\n",
    "chars, counts = zip(*counter.most_common())\n",
    "indices = np.arange(len(counts))\n",
    "\n",
    "plt.figure(figsize=(14, 3))\n",
    "plt.bar(indices, counts, 0.8)\n",
    "plt.xticks(indices, chars);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We cut the dataset in fake sentences of fixed length. This could be greatly improved by using a sequence tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nb train sequences: 334597\n",
      "nb test sequences: 11150\n"
     ]
    }
   ],
   "source": [
    "max_length = 40\n",
    "step = 3\n",
    "\n",
    "\n",
    "def make_sequences(text, max_length=max_length, step=step):\n",
    "    sequences = []\n",
    "    next_chars = []\n",
    "    for i in range(0, len(text) - max_length, step):\n",
    "        sequences.append(text[i: i + max_length])\n",
    "        next_chars.append(text[i + max_length])\n",
    "    return sequences, next_chars    \n",
    "\n",
    "\n",
    "sequences, next_chars = make_sequences(train_text)\n",
    "sequences_test, next_chars_test = make_sequences(test_text, step=10)\n",
    "\n",
    "num_sequences = len(sequences)\n",
    "print('nb train sequences:', num_sequences)\n",
    "print('nb test sequences:', len(sequences_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.utils import shuffle\n",
    "\n",
    "sequences, next_chars = shuffle(sequences, next_chars,\n",
    "                                random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"ragg'd progeny, thou shouldst not scape \""
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequences[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'m'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next_chars[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "use_gpu = torch.cuda.is_available\n",
    "def gpu(tensor, gpu=use_gpu):\n",
    "    if gpu:\n",
    "        return tensor.cuda()\n",
    "    else:\n",
    "        return tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each sentence will be turned into a tensor, specifically a `LongTensor` (used for integer values), by looping through the characters of the string and looking up the index of each character in `char_to_indices`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([11, 12, 13, 14, 15, 16], device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "# Turn string into list of longs\n",
    "def char_to_tensor(string):\n",
    "    tensor = torch.zeros(len(string)).long()\n",
    "    for c in range(len(string)):\n",
    "        tensor[c] = char_to_indices[string[c]]\n",
    "    return gpu(tensor)\n",
    "\n",
    "print(char_to_tensor('abcdef'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Measuring per-character perplexity\n",
    "\n",
    "The NLP community measures the quality of probabilistic model using [perplexity](https://en.wikipedia.org/wiki/Perplexity).\n",
    "\n",
    "In practice perplexity is just a base 2 exponentiation of the average negative log2 likelihoods:\n",
    "\n",
    "$$perplexity_\\theta = 2^{-\\frac{1}{n} \\sum_{i=1}^{n} log_2 (p_\\theta(x_i))}$$\n",
    "\n",
    "**Note**: here we define the **per-character perplexity** (because our model naturally makes per-character predictions). **It is more common to report per-word perplexity**. Note that this is not as easy to compute the per-world perplexity as we would need to tokenize the strings into a sequence of words and discard whitespace and punctuation character predictions. In practice the whitespace character is the most frequent character by far making our naive per-character perplexity lower than it should be if we ignored those.\n",
    "\n",
    "**Exercise**: implement a Python function that computes the per-character perplexity with model predicted probabilities `y_pred` and `indice_true` for the ground truth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def perplexity(indice_true, y_pred):\n",
    "    \"\"\"Compute the perplexity of model predictions.\n",
    "    \n",
    "    indice_true is ground truth: indice of character.\n",
    "    y_pred is predicted likelihoods for each class.\n",
    "    \n",
    "    2 ** -mean(log2(p))\n",
    "    \"\"\"\n",
    "    #\n",
    "    # your CODE here\n",
    "    #"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.2598816041129843"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indice_true = [1,2]\n",
    "\n",
    "y_pred = torch.Tensor([[0.1, 0.9, 0.0],[0.1,0.2,0.7]])\n",
    "\n",
    "perplexity(indice_true, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build the Model\n",
    "\n",
    "This model is taken from PyTorch tutorial. It will take as input the character for step $t_{-1}$ and is expected to output the next character $t$. There are three layers - one linear layer that encodes the input character into an internal state, one GRU layer (which may itself have multiple layers) that operates on that internal state and a hidden state, and a decoder layer that outputs the probability distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size, n_layers=1):\n",
    "        super(RNN, self).__init__()\n",
    "        self.input_size = input_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.output_size = output_size\n",
    "        self.n_layers = n_layers\n",
    "        \n",
    "        self.encoder = nn.Embedding(input_size, hidden_size)\n",
    "        self.gru = nn.GRU(hidden_size, hidden_size, n_layers)\n",
    "        self.decoder = nn.Linear(hidden_size, output_size)\n",
    "    \n",
    "    def forward(self, input, hidden):\n",
    "        input = self.encoder(input.view(1, -1))\n",
    "        output, hidden = self.gru(input.view(1, 1, -1), hidden)\n",
    "        output = self.decoder(output.view(1, -1))\n",
    "        return output, hidden\n",
    "\n",
    "    def init_hidden(self):\n",
    "        return torch.zeros(self.n_layers, 1, self.hidden_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "hidden_size = 100\n",
    "n_layers = 1\n",
    "lr = 0.005\n",
    "\n",
    "decoder = gpu(RNN(n_characters, hidden_size, n_characters, n_layers))\n",
    "decoder_optimizer = torch.optim.Adam(decoder.parameters(), lr=lr)\n",
    "criterion = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the network on the first sentence in the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'gremio: good morrow, neighbour baptista.'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequences_test[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([40, 1])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(tensor([[ 0.1083,  0.1114,  0.1071, -0.0732, -0.2118, -0.0474,  0.0292,  0.0363,\n",
       "           0.1222, -0.1419,  0.0524,  0.1270, -0.1002, -0.1856,  0.0808, -0.2566,\n",
       "          -0.1205, -0.1808,  0.0370,  0.0575, -0.0189,  0.3111, -0.0011, -0.1940,\n",
       "           0.1401,  0.0107, -0.0616, -0.1133, -0.0455,  0.0839, -0.0672,  0.1398,\n",
       "           0.1174, -0.1363, -0.2937, -0.3613,  0.2529]],\n",
       "        device='cuda:0', grad_fn=<ThAddmmBackward>),\n",
       " tensor([[[ 0.0135, -0.2500,  0.1071,  0.0698,  0.0703, -0.0331,  0.1751,\n",
       "            0.3541, -0.2664, -0.3061,  0.1968,  0.0157, -0.0211, -0.3073,\n",
       "           -0.2975, -0.6584, -0.2380,  0.1359,  0.0153, -0.0541,  0.0674,\n",
       "           -0.2319, -0.2751,  0.2931, -0.1641, -0.2029, -0.3467,  0.3329,\n",
       "           -0.3563, -0.5393,  0.0339,  0.0345, -0.1413, -0.4366, -0.1044,\n",
       "            0.0103, -0.3574, -0.0400,  0.0318, -0.0649, -0.0880, -0.3485,\n",
       "           -0.0491,  0.2123,  0.0846, -0.3381,  0.1759, -0.0963, -0.1125,\n",
       "            0.0649, -0.1663, -0.3903,  0.0392,  0.0796,  0.1947,  0.2743,\n",
       "           -0.0693, -0.1521, -0.0736,  0.2158,  0.1615,  0.0928, -0.2270,\n",
       "           -0.1359, -0.3696,  0.0534,  0.3476, -0.2685,  0.3132,  0.2819,\n",
       "           -0.0633, -0.0350,  0.1652, -0.0681,  0.1384,  0.1450, -0.3687,\n",
       "            0.0655,  0.1546, -0.0629, -0.0995,  0.2379,  0.1681, -0.4471,\n",
       "           -0.1682, -0.0493, -0.2597,  0.3109, -0.0647,  0.0140, -0.0256,\n",
       "            0.2183,  0.2599,  0.0962,  0.1381,  0.3037, -0.2771, -0.2290,\n",
       "            0.0299,  0.5641]]], device='cuda:0', grad_fn=<CudnnRnnBackward>))"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating\n",
    "\n",
    "To evaluate the network we will feed one character at a time, use the outputs of the network as a probability distribution for the next character, and repeat. To start generation we pass a priming string to start building up the hidden state, from which we then generate one character at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(prime_str='t', predict_len=100, temperature=0.8):\n",
    "    hidden = gpu(decoder.init_hidden())\n",
    "    prime_input = char_to_tensor(prime_str)\n",
    "    predicted = prime_str\n",
    "\n",
    "    # Use priming string to \"build up\" hidden state\n",
    "    for p in range(len(prime_str) - 1):\n",
    "        _, hidden = decoder(prime_input[p], hidden)\n",
    "    inp = prime_input[-1]\n",
    "    \n",
    "    for p in range(predict_len):\n",
    "        output, hidden = decoder(inp, hidden)\n",
    "        \n",
    "        # Sample from the network as a multinomial distribution\n",
    "        output_dist = output.data.view(-1).div(temperature).exp()\n",
    "        top_i = torch.multinomial(output_dist, 1)[0]\n",
    "        \n",
    "        # Add predicted character to string and use as next input\n",
    "        predicted_char = indices_to_char[top_i.item()]#all_characters[top_i]\n",
    "        predicted += predicted_char\n",
    "        inp = char_to_tensor(predicted_char)\n",
    "\n",
    "    return predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'whatoe,tf,b&'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate('wh',10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training\n",
    "\n",
    "form PyTorch turorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time, math\n",
    "\n",
    "def time_since(since):\n",
    "    s = time.time() - since\n",
    "    m = math.floor(s / 60)\n",
    "    s -= m * 60\n",
    "    return '%dm %ds' % (m, s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(inp, target):\n",
    "    \n",
    "    hidden = gpu(decoder.init_hidden())\n",
    "    decoder.zero_grad()\n",
    "    loss = 0\n",
    "    \n",
    "    for c in range(max_length-1):\n",
    "        output, hidden = decoder(inp[c], hidden)\n",
    "        loss += criterion(output, target[c].unsqueeze(0))\n",
    "\n",
    "    loss.backward()\n",
    "    decoder_optimizer.step()\n",
    "\n",
    "    return loss.item() / max_length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exercise: write the training loop. There is a slight modification copmared to the PyTorch tutorial because we did split the dataset in train and test... compute the perplexity on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0m 4s (epoch: 1 4%) 2.4137]\n",
      "wh hagthe, lar ed thomepeind be prbrove bond bitin hy ot bort ken heard athe and alnd teer;  pite hous  \n",
      "\n",
      "[0m 8s (epoch: 1 9%) 2.3177]\n",
      "wh anere suce where there or i have ar therer.  at shat manots gour: of ay ther owe withe padle's thear \n",
      "\n",
      "[0m 13s (epoch: 1 14%) 2.1824]\n",
      "wh son dee min the sit ar, a mand and and thi so and wourl; the sed lever ploving and not mope and any  \n",
      "\n",
      "[0m 17s (epoch: 1 19%) 2.0682]\n",
      "wh dussphio: of haene the wil dom digh the mave is heaw bot hit suck hougk than restw is fat and way, f \n",
      "\n",
      "[0m 21s (epoch: 1 24%) 2.1304]\n",
      "wh the wheriom dove the of dort dearung faver, hivere deard there thes fall; i hik: wit writ thing.  od \n",
      "\n",
      "[0m 26s (epoch: 1 29%) 1.9171]\n",
      "wh all of i wran, i have duege youd; coushill an a known posdes rom: avere the that the a haust you sli \n",
      "\n",
      "[0m 30s (epoch: 1 34%) 1.8630]\n",
      "wh the and ye whestere carenintal, wher to mre your foor omors: and your carce thist ond shon as the da \n",
      "\n",
      "[0m 34s (epoch: 1 39%) 1.7078]\n",
      "wh time, this hath bet cangoure sight not chasst not wo hath caing youe ware .: lorning lig, i with tha \n",
      "\n",
      "[0m 38s (epoch: 1 44%) 1.6946]\n",
      "wh ed as fronghirm thin stheme of i the him us the an my fithe so mainn, bustire shall mur mue of ustau \n",
      "\n",
      "[0m 43s (epoch: 1 49%) 1.8850]\n",
      "wh eay: sure, wer'd de thy as sut descoee, now, you heas strould the quell, browt seadl now, loke, she  \n",
      "\n",
      "[0m 47s (epoch: 1 54%) 1.6813]\n",
      "wh thy huself of you, his chat the lelive hath how coppyorth quren, frown for of faor heard this us the \n",
      "\n",
      "[0m 51s (epoch: 1 59%) 2.1210]\n",
      "wh i godeas sathage, wis my sirle oblastesest sus: i a bace browhing us sears i pice a and pare: thee s \n",
      "\n",
      "[0m 56s (epoch: 1 64%) 1.9379]\n",
      "wh i le not him, and fard theme searn have who him dand the will ere im amboris: aste the princa: the w \n",
      "\n",
      "[1m 0s (epoch: 1 69%) 1.8944]\n",
      "wh thee mort cove fromo to theere? are, love, tham to mutherforething to the more the plave ame of i th \n",
      "\n",
      "[1m 4s (epoch: 1 74%) 1.9919]\n",
      "wh thell, not this?  frowre i that i be adre to sthing vasinet mores foor mefore more that the stoo? no \n",
      "\n",
      "[1m 9s (epoch: 1 79%) 1.7440]\n",
      "wh how to gentres bentither chprown luch whe pitly grocusire and on prvine, corh look, i are the prigac \n",
      "\n",
      "[1m 13s (epoch: 1 84%) 1.8320]\n",
      "wh comich supt the and lord: ay what forh, we sopt though quood houll in of prist from and will his fro \n",
      "\n",
      "[1m 17s (epoch: 1 89%) 2.1501]\n",
      "wh to tro me thi warrwick?  muart i hall hing prire soagusorn doo, three haster with my art on hencess  \n",
      "\n",
      "[1m 22s (epoch: 1 94%) 1.8411]\n",
      "wh you shatal, for that thim, has, and the sting, thin; your what your hath me to shall thy our shalbna \n",
      "\n",
      "[1m 26s (epoch: 1 99%) 2.0214]\n",
      "wh i wow fir i am streent. if sham haring mame to me to omense! think pencound; there of hand me come m \n",
      "\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 1\n",
    "print_every = 100\n",
    "plot_every = 10\n",
    "n_pass = 2000\n",
    "\n",
    "start = time.time()\n",
    "all_losses = []\n",
    "loss_avg = 0\n",
    "\n",
    "for epoch in range(1, n_epochs + 1):\n",
    "    #\n",
    "    # your CODE here\n",
    "    #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Plotting the Training Losses\n",
    "\n",
    "Plotting the historical loss from all_losses shows the network learning:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f5c344ac320>]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnXecI1eV739HUimnznmmJ9gznuixx/YY2+BAshdjTFrCmrhrWMx7mLAP2AC8ZdllCV7SGtaLwcAzYFgb1ktywgFnz4wn55w6B6mV031/VN1SSV1Sq3u61S3N+X4+85luqSTdLkm/e+p3zj2XhBBgGIZh6gvLfA+AYRiGmX1Y3BmGYeoQFneGYZg6hMWdYRimDmFxZxiGqUNY3BmGYeoQFneGYZg6hMWdYRimDmFxZxiGqUNs8/XCzc3Nore3d75enmEYpibZsmXLsBCiZarj5k3ce3t7sXnz5vl6eYZhmJqEiI5XchzbMgzDMHUIizvDMEwdwuLOMAxTh7C4MwzD1CEs7gzDMHUIizvDMEwdwuLOMAxTh9ScuO/vn8DXH96PkUhyvofCMAyzYKk5cT8yFMG3/3gIQyzuDMMwJak5cXco6pAT6dw8j4RhGGbhUnPi7rRZAQDJdHaeR8IwDLNwqTlxdyiquCcyHLkzDMOUovbE3SZtGY7cGYZhSlFz4u7UIvckR+4MwzAlqUFx58idYRhmKmpQ3DmhyjAMMxU1J+7Sc2dbhmEYpjQ1J+4ycmdbhmEYpjQ1J+6K1QKrhXgRE8MwTBlqTtwB1ZpJZjhyZxiGKUVNirtTsXLkzjAMU4YpxZ2InET0IhFtJ6LdRPR/TY5xENF9RHSIiF4got65GKzEabOw584wDFOGSiL3JIBrhRDrAVwI4PVEtKnomA8CGBNCLAfwbwD+dXaHWYhDsXL7AYZhmDJMKe5CJaL9qmj/RNFhNwH4kfbzfwG4joho1kZZhMNm4Tp3hmGYMlTkuRORlYi2ARgE8IgQ4oWiQ7oAnAQAIUQGQAhA02wO1IiTI3eGYZiyVCTuQoisEOJCAN0ALiWiNTN5MSK6lYg2E9HmoaGhmTwFADVyZ8+dYRimNNOqlhFCjAN4HMDri+46DaAHAIjIBiAAYMTk8XcJITYKITa2tLTMbMRQI3deocowDFOaSqplWogoqP3sAvAaAPuKDnsQwHu1n98K4I9CiGJfftZwKuy5MwzDlMNWwTEdAH5ERFaok8EvhBC/IaJ/BLBZCPEggLsB/ISIDgEYBfCOORsxZJ07izvDMEwpphR3IcQOABtMbv+c4ecEgLfN7tBKo65QZVuGYRimFDW8QpUjd4ZhmFLUsLhz5M4wDFOKmhR3h82CRCaLOczZMgzD1DQ1Ke5OxQohgHSWxZ1hGMaMmhR3uRtTgtv+MgzDmFKb4s67MTEMw5SlJsXdKfdR5aQqwzCMKbUp7lrkzrsxMQzDmFOT4q577hy5MwzDmFKT4s6RO8MwTHlqWtw5cmcYhjGnRsVd2jIcuTMMw5hRk+LusElbhiN3hmEYM2pS3DlyZxiGKU+Nijt77gzDMOWoSXHPl0Jy5M4wDGNGTYp7vhSSI3eGYRgzalLcOXJnGIYpT02KOxHBrvV0ZxiGYSZTk+IOqM3DuHEYwzCMObUr7oqV2w8wDMOUoGbF3aFYuBSSYRimBFOKOxH1ENHjRLSHiHYT0cdMjgkQ0f8Q0XbtmPfPzXDzOG0cuTMMw5TCVsExGQCfFEJsJSIfgC1E9IgQYo/hmNsA7BFC3EhELQD2E9G9QojUXAwaANx2K6JJFneGYRgzpozchRB9Qoit2s8TAPYC6Co+DICPiAiAF8Ao1ElhzvC7FEwk0nP5EgzDMDXLtDx3IuoFsAHAC0V3fQfABQDOANgJ4GNCiEmGOBHdSkSbiWjz0NDQjAYs8TsVhOIs7gzDMGZULO5E5AVwP4DbhRDhortfB2AbgE4AFwL4DhH5i59DCHGXEGKjEGJjS0vLWQxbjdzDiTm9OGAYhqlZKhJ3IlKgCvu9QogHTA55P4AHhMohAEcBrJy9YU7G77IhzJE7wzCMKZVUyxCAuwHsFULcUeKwEwCu045vA7ACwJHZGqQZfqeCZCbHLQgYhmFMqKRa5goAtwDYSUTbtNv+FsAiABBCfA/AFwHcQ0Q7ARCATwshhudgvDoBlwIACCfSeiMxhmEYRmVKcRdCPA1VsMsdcwbAa2drUJXgl+Iez6DVV81XZhiGWfjU7ApVv1Odl7hihmEYZjI1K+5GW4ZhGIYppGbFPW/LsLgzDMMUU7vi7mRxZxiGKUXtirtL9dx5IRPDMMxkalbcHTYrnIqFI3eGYRgTalbcAe4vwzAMU4qaFveAS+FqGYZhGBNqWtz9LgXhOHvuDMMwxdS2uDttbMswDMOYUNPizrYMwzCMOTUt7n4XJ1QZhmHMqG1xdyoIx9MQQsz3UBiGYRYUNS3uAZeCnACiKe7pzjAMY6SmxV2uUmVrhmEYppDaFnfuL8MwDGNKTYt7g8cOABiNpuZ5JAzDMAuLmhb3dr8TANAfSszzSBiGYRYWtS3uAU3cwyzuDMMwRmpa3J2KFUG3wpE7wzBMETUt7oBqzfSxuDMMwxQwpbgTUQ8RPU5Ee4hoNxF9rMRxVxPRNu2YJ2d/qOa0B5zoD8er9XIMwzA1ga2CYzIAPimE2EpEPgBbiOgRIcQeeQARBQHcCeD1QogTRNQ6R+OdREfAiV2nw9V6OYZhmJpgyshdCNEnhNiq/TwBYC+ArqLD3gXgASHECe24wdkeaCna/S4MR5JIZXLVekmGYZgFz7Q8dyLqBbABwAtFd50PoIGIniCiLUT0ntkZ3tS0BxwAgAGumGEYhtGpxJYBABCRF8D9AG4XQhT7IDYAFwO4DoALwHNE9LwQ4kDRc9wK4FYAWLRo0dmMW6c94AKgintPo3tWnpNhGKbWqShyJyIFqrDfK4R4wOSQUwAeEkJEhRDDAJ4CsL74ICHEXUKIjUKIjS0tLWczbp0OrdadK2YYhmHyVFItQwDuBrBXCHFHicP+G8CVRGQjIjeAy6B683NOG69SZRiGmUQltswVAG4BsJOItmm3/S2ARQAghPieEGIvEf0BwA4AOQDfF0LsmosBF+N32uC2WzlyZxiGMTCluAshngZAFRz3VQBfnY1BTQciQnvAyQlVhmEYAzW/QhUAzm/14dG9A3h0z8B8D4VhGGZBUBfi/k83r8GKdh8+9P+2YH//xHwPh2EYZt6pC3Fv9jpwx9vXI5sT2NvHq1UZhmHqQtwBoMWnVs0MR5LzPBKGYZj5p27E3e+0wW6zYGiCxZ1hGKZuxJ2I0OJ1YIgjd4ZhmPoRdwBo9jkwHOH9VBmGYepK3Fu8drZlGIZhUGfi3ux1cEKVYRgGdSbuLT4HRiJJZHNivofCMAwzr9SVuDd7HcgJYCzGvjvDMOc2dSfuANh3ZxjmnKeuxL3Fp4o7++4Mw5zr1JW4N3vtAFjcGYZh6krcZeQ+lS2TyuR4AmAYpq6pK3H3Omxw2CxTLmT68XPH8Jo7noQQXFXDMEx9UlfiTkRo9jqmjNz7QgmMxdKIpbJVGhnDMEx1qStxB1RrZirLJZbKAAAmEplqDIlhGKbq1J24VxK5y4g9kkxXY0gMwzBVp+7EvcGtYDxWXrSluHPkzjBMvVJ34h50KxiPl0+osi3DMEy9U4fibkcinUMiXTpZmrdlMnh4dz+u+doTSGVy1RoiwzDMnDOluBNRDxE9TkR7iGg3EX2szLGXEFGGiN46u8OsnKBbAYCy1kxcinsig12nQzg6HOV+NAzD1BWVRO4ZAJ8UQqwCsAnAbUS0qvggIrIC+FcAD8/uEKdHg1tdpVrOmolqtkw4kcaYNgmE45xcZRimfphS3IUQfUKIrdrPEwD2AugyOfR/AbgfwOCsjnCaBF1q5D4WrSByT2b0iD2cYHFnGKZ+mJbnTkS9ADYAeKHo9i4ANwP47mwNbKYEtcg9VCZyN1bLjOuROydXGYapHyoWdyLyQo3MbxdChIvu/gaATwshymYliehWItpMRJuHhoamP9oKkJ77mMFzf+bQsB6t53Iin1BNZHT7hiN3hmHqiYrEnYgUqMJ+rxDiAZNDNgL4OREdA/BWAHcS0ZuKDxJC3CWE2CiE2NjS0nIWwy6N7rlr4n5wYALv/v4L+OmLJwAAiUy+iiaSzOj2DXvuDMPUE7apDiAiAnA3gL1CiDvMjhFCLDEcfw+A3wghfj1bg5wOTsUCu82iR+RPHlCvELafHAeAgn4y4UQa47rnzrYMwzD1w5TiDuAKALcA2ElE27Tb/hbAIgAQQnxvjsY2I4hIXaWqReRPHRwGAOw6HQIAxJJ5cR+LpRDVxJ4jd4Zh6okpxV0I8TQAqvQJhRDvO5sBzQZBlx3j8RQS6SxeODICh82CI8NRTCTSiKXVCN1CwKmxuP4Y9twZhqkn6m6FKqAmVcdiabx0bBTJTA7vvHQRAGD3mbBuy7T4HAULnbhahmGYeqJuxT0US+NPB4dht1rwwSvVlMDOUyHdlmn1OQsew5E7wzD1RF2Ke4PbjrFYCi8eHcX6ngB6Gt3oCDix83RIbxrW5nfoxzd77ey5MwxTV9SluAfcCsZiKew+E8LFixsBAGu7Ath1OoS41lCs1Z+P3Bc1urlahmGYuqIuxb3BbUc6K5DOCly0KAgAWN7qxYnRmN7mt9WXj9wXNbo5cmcYpq6oS3GX/WUA4KLFDQCANr8TmZzA6fG4/jsAKFZCe8CFcCLNG2YzDFM31Ke4a6tUFze50exVI3TpsR8bjhb8HnTbEXApSGcFEmnu6c4wTH1Qp+KuRu4XL2rQb5Me+9HhKBQr6RNA0KXA71LL/blihmGYeqEuxb3Zqwq3tGSAvA1zbCQKt90Gn0MV9Aa3HX6nOhmw784wTL1QSfuBmmNZixd3vvsiXLuyVb+tRbNnEukcGtx2+DRBD7oV+DWPniN3hmHqhbqM3IkIN6ztgFOx6rfZbRY0edSI3m23wutU57WgW4Ff+3mhrFL95qMHcccjB+Z7GAzD1DB1Ke6lkL67226Dx26F3WpBk9ex4CL3Jw4M4vF987qhFcMwNU5d2jKlaPM7sLcPcNmtICL84H2X4Pw2L9SuxgvHc4+nskhmuHKHYZiZc26Ju9ZPxmNX7Zorz2sGACS0VauhBSLusVRWb5PAMAwzE84pW0bWtrvthXOaU7HCYbMsmBYEsVRGX0nLMAwzE84pcZeeu8tunXRfs9eBwXCi2kMyJabZMim2ZhiGmSHnlLjLWnePibj3NLpwYjRW7SFNomAD7yRH7wzDzIxzTNxVW8Zln5xqWNzoWRDibtzAe2KBVO8wDFN7nGPiLkshJ0fui5rcGI6k9Gg5lxM4qvWhqSbGDbzZd2cYZqacU+Le4nXgrRd34yqtSsbI4iY3AODkaAzZnMCnfrkd13ztiaoLvHEDbxZ3hmFmyjlVCmmxEL72tvWm9y1u9AAAjo/EcO8Lx/HAy6cBAIcHI1jS7KnaGOUG3gB77gzDzJwpI3ci6iGix4loDxHtJqKPmRzzbiLaQUQ7iehZIjJX0AXMokY1ct/TF8Z9L53E61a3AQBOjc2tDx9LZfCrl08hnc1pv+cj90iSPXeGYWZGJbZMBsAnhRCrAGwCcBsRrSo65iiAVwkh1gL4IoC7ZneYc0/ArSDgUvDLzSeRzgq89xW9cCoWnBqLoz+UwAfveQmDE7NfKvmHXf34+H3b8ZF7tyKZybItwzDMrDCluAsh+oQQW7WfJwDsBdBVdMyzQogx7dfnAXTP9kCrweImN/pCCXgdNlzS24juBjdOjcXx5IFBPLZvEP+15dRZv8bvdvbhwMCE/rv09B/ZM4Av/35fwcpUFneGYWbKtBKqRNQLYAOAF8oc9kEAv5/5kOYPac1csbwJitWC7gYXTo7FsK9fFeMHt505q+fP5gRuv28b/vOpI/ptR4ejWNzkxqW9jdh9JjxltYwQAv/x5OE5uYpgGKZ+qFjcicgL4H4AtwshwiWOuQaquH+6xP23EtFmIto8NDQ0k/HOKbJi5uoVah/4Hi1y36+J+77+Cf3nmXB6LI5UJod+w0rY4yMxLG7yoMGjIBRLT+m5nwkl8C+/34ff7uiDEALfePQAd5BkGGYSFYk7ESlQhf1eIcQDJY5ZB+D7AG4SQoyYHSOEuEsIsVEIsbGlpWWmY54z1nUH4VKsuEYT9+4GF0LxNHacCuG6la2wEPDg9tMzfv7DQxEAQH9IFXchBI4NR9Hb5EbQZcd4PKXbMg1uxTRyj2i3jcfSGI+l8Y1HD+L997yEbz92cMbjYhim/qikWoYA3A1grxDijhLHLALwAIBbhBA1u8vEa1e1Ycs/vBrtAXWxU3eDGslHkhm8YnkzLl/WhEf2DOjH53IC3//TEYRilVW16OKuRe6j0RQmkhn0NnkQdCsYN0TurT6nLuRGZDQfiqcxFksBULcVvOPRA0gaVrcyDHNuU0nkfgWAWwBcS0TbtH83ENGHiejD2jGfA9AE4E7t/s1zNeC5hIgKOkZ2N7j0n1e2+3DF8mYcGIhgOJIEoJZN/tNv9+Inzx+DEAK/3dFXdsMPKe4TiQxiqQyOjahllr3NbgTcCpKZHEajKditFgRLRe5aNc14LIVxrUXxZUuaIAQwGE6e5RmonGxO6OeBYZiFRyXVMk8LIUgIsU4IcaH273dCiO8JIb6nHfOXQogGw/0b537oc0+PlmAFgBXtPmxa2gQAePHoKIC8vfLIngG8eHQUt/10K370zLGSz3d4ML/atT+UwDGtUqa3yYMGt7oF4JnxOFx2K3xOBRMmi5h0WyaexrgWua9s9wEA+kLVS7Lev/UUXvWVxxFP8dUCwyxEzqn2A9Olwa3Abbei2WtHs9eBtV0BuO1WPH9ETSn0afbK9lMhfONR1fN+5vBwyec7PBRBT6N6NdAfTuD4SBQWUu2foLbV35lQHB67FT6nzTShKm+TnjsArOzw689ZLU6OxhBNZTl6Z5gFCot7GYgIvU0erOoMAAAUqwWX9Dbq4t4fiuvHPndkBHabBVuPjyORzuLXL5/W+8OfGImhP5TASDSFK5apfW0GwgkcHYmhu8ENu82CgFsV977xhBa520xtGXmb6rlr4q5F7gNVjNzlloTS92cYZmHB4j4F337XBvzzzWv03zctbdJ99/5QEh0Bp9575hOvOR+pbA7/9ugB3H7fNvz0xRPI5gRu+Naf8KZ/fwYA8Irlqrj3h5I4PhLVyy+DLtWWGYmm4HHY4HXYEElkIIQoGE/U6LnHUrAQ0BV0waVYqxq5y12rRqMs7gyzEGFxn4JlLV69agYALl3SCADYcnwM/eE4OgJOfOiVS/HOS3twy6bFsFkI//Gkukjp5Ggc/eEEIsmMLrzruwPwOmw4NhzF3r4wVmmWSlCL3AHApaieeyYnkEgX7sZkrJYZjaYQcCmwWAjtAWfF4h5OpPEPv951Vvu01kPk/vTBYdz+85fnexgV8dzhEfx+Z998D4OpIVjcp4m0QA70T6AvlEB7wIl3XLoI//LmdfA4bLiwJwgAUKyEU2MxnNQ2APmrq5bgbRd3o6fBjTa/A7/f1Yd0VuCq89R6f5lQBdR+816nWrUzUeS76/3mBXByLK4/rs3vqNiWeenoKH7y/HFsOzE+09OgVwWNRWu3udmfDg7h19vOIJNd+NsZ3vnEIXzpd3vnexhMDXFOtfydDTwOG3oaXdg/MIH+UAJXn99acP/bL+mB12mDz6lg6/ExXdzfddli3b5pDzhxeCgKh82Cjb0NAACnYoHdZkEqk4PbYYPPoYl7IoNWX/75jT788ZEoGj2quLf7ndh8fAyVICcIs2qcSgnH1cfOZuQuhMCpsTjaA04o1rmPO+Tfn8jk4K3C650N/aEEBsIJZHMCVgvN93CYGmBhf6IXKOe3+rD1+BhiqSzaA46C+96+sQf3vP9S9Da5tYqYGIiAzqBTP0buCHXpkkY4FXVXKCLSK2bcippQBTBpIVPUIMinjJF7wInBcHKSR2+GnCCkp//CkRHkclM/zogeuc+SuG85PoYbvvU0rvrK4/j5Sydn5TmNCCHw748fwvGRfDmqPJfJ9MIv5xwIJ5DO8tqCeCpbE1daCwEW9xlwfrsPZzQLpD3gMj2mu8GFbE7gxWOjaPc74bDlt/Zr18S9eEco6bvLhCowuXY9kszArkWZ2ZzQH9PudyKVzVWU4JSiFklmsPtMGH9+1/N4cPv0mqLpnnsFtsyzh4bx3GHTjhQ6/7XlFI4OR2C1EM6Mx8seOxNOjcXx1Yf24wP3vKRfuUQNkXu1mUikkarwdeOprJ7APj0H56ZWEELg+m8+hf/7P3vmeyg1AYv7DFjRlvdJOgJO02NkEvblE2PoMSRkjfdJv10S1KJwl92KVZ1+dAVd+PyDu9BnKLmcSGTQZVg5K6ts5IRRSVJVt2USaQxpkeDvppGsy2RziGqLl6aaTI4MRfDBH23G5x/cVfa4UDyFrqALAZcyJxuDy3EeHori73+1E0D+PMxH5P6mf38G33yssk4dxvd0Lia+WuH4SAzHRmK4b/NJjJzjVzCVwOI+A843iLsU1WKkoKezoqCNAQC8aUMnfvyBS3GBVikjKbRlFPzgfZcgmszib365Qz8mkswUPF+DFrm3aZPMwHTEPZnRI/AnDwwVWD4AcGgwgoSJ8IUNVlE5WyaTzeETv9iOeDqLw0NR0+fSnyeaRoPbDr/Tpvv5s4kU91Udfvx+Vz+AfFlpcUVSNTg5Fseu06bNVScxUAVxF0Lgdzv7FrTl8dIxdWV4KpPDT184MaPnSGayuOXuF/DyicryU7UMi/sMWNri0ZNarX6H6THtASdk3qu7sTByd9tteOX5k7tiSovFrVkyK9p9eNdli/Di0VH9Ej6azKAraIjcPUWRe2jqiCZi8NyluCczOTyxP9+GeTCcwOu/8RTe8t1nJ1kB8jF2m6WsuD+2bxDbTo7jhrXtyOaE3hffjLFYCkG3Ar9LKdufZ6aMaOK+utOPZCaHRDqbj9yr3HAtlckhlcnpyfapKBT3uVnLsP1UCB+5dyuePLDwWnFLNh8bQ9Ct4KrzmvHj54/rW1NOh2PDMfzp4HDBZ73ajMdSFeXGzhYW9xngVKxY3ORGs9de4KUbsdssuuD2NJj78sVIW8Ztzz/n+u4gUtkc9vWHIYRAJJlBg8euHyOj/RafAxbKR3Y/fOYoNmuRTjERg+cuo/CgW8HvduWtmX39E8hogvyeuwv3ZpHiu6jRjbFouuQH9ZlDw3DbrfjUa1cAAHafCZX820PxNIJuO3xOmz55zCajUXXS69UqliYSGf08JNI5vHh0FO/74YsV++Bng7xCOjUWryiRLXsYdQVdc+a5D02o52ckkkIonsZr7ngSu06Xfr/mg5eOj2Lj4gbcvKELQxPJguR4pZweVyfUSifW2UYIgSu+/Ed86bdzX9bK4j5DNi1twmqtLUEppLfeUxS5l0KP3I3i3qO+xvZTISQzOaSzAl6HTRd1WS2jWC1Y1uLFnr4wYqkMvvibPfjhs8dMXyfvuauRu8NmwZs3dOMPu/pxUNsC8NCg2sHyPZcvnmSpSNukt8mNVDZXsMGIkWcODeOS3kYsafbA77Rh95nSNsRYLIUGtwK/07wbZiU8tncAf9Asl2JGoikoVtKrlsKJdL5aJpPFS8dG8cT+oYItEOcKef5T2RwGKthRayCchMduxflt3jmzZeQV2Hg8hWPDURwcjJSdjKvNSCSJI0NRbOxt1Ftyz6QL6ukx9fydnOON70sRjmcQTWX1v2EuYXGfIf900xr88H2XlD2mW2sSVrG4u2Tknl9+0BV0ocljx/aT47oo+Jw2BDRRN65sXdcdxI5TIWw/GUJOAIcGIvp9Qggc0VoO65F7IoNwIg2/S8FHr10Ot92Kf9YWyhwaiiDoVrC2S51cjKKSj9zVKNgsqToQTuDwUBRXLG8CEWF1Z6CkuCfSWSTSOQTddvidhbZMKJ7GE/sr22nqO48fwtcf3m9631g0hUaPHQFtUjT2zk+kc3p3yz1lJqCzQQiBD97zEv6wq08//4DadwhQz+F3/njQ1PMeCCfQ5neiM+iatrj3heKTcilmjGnv4VgsjVFN6EtN2vOBXMOxcXEDWn2auE9MX9xPaefv5Kj6/85TobNaqT1d5JWX0VqdK1jcZ4jFQrBMsZhkdWcATR57yaRrMWaROxFhfU8Q20+O619Sr8OGgMtW8BhAjfKHI0n8XrNXjgxHdLF4ZM8ArrvjSRwfieqe+0Qyg1A8Db/ThkaPHf/72vPw+P4hPH9kBIcGI1je4kWHVuppLMmUtklvszppjZtsVvKs1h3zFVqjtNWdfuzrC5uKl3x80K1otkz+y/bzF0/gfT98qaI9Y4cjSRwdjpp6saPRFBo9Dl3c+w1/TzKTRVy7MtnTNzfifnwkhsf2DeK5wyOF4q7ZA9989AC+9vAB0wmw3yDuY7H0tMTozXc+i397ZOqqHCno47G0LvRm4n5wYKJkYjyWyuCh3eZXTpKTozGs+8JD074qkFeSqzr9ep5rJvsIy8i9P5zAYDiBm+98Bt98tHq7mMnJuZPFvbZ57+WL8fjfXF3xisLVnX4sbnJjWYu34PZ13QEcGoroguRx2PQo39i2YF232vrg/i2nAKiVOsc18dh2chxCqB/ufJ17GuF4Rhe8Wy5fDJdixe929uHwYATLW726jWEWuS9u0iJ3k6TqM4dGEHQreu+c1V1qIvPw0GSfVFoCDW47/C4F8XRWF2g5qRwciGDHqXG8+/vPl/SdhydSyOSEqRc7Ek2hyaNeGRT/Pcl0Thf3ubIitmiRZ9jg9QOq2A1OJPAzbeHWkEk0OhBW21zIaK/SpOp4LIW+UAJHh6f2pqWgj8dS+pVYccQfS2XwZ99+Gj9+7pjpczy47Qw+9JMtk87/gYEJrP38QzgyFMG2k+MIJzLYWqb1hdwE3jjusWgKLsUKt11dve1ULKbnaiqMn53f7uxDJifw4PYz017EN1POhFjc6wKb1aKLSSUsbvLgyb+5ZpIft74nCCHUtsIA4HPYEHQrsFstBVH+BR0+KFZCNJXFshZVeA9q1oysVBkpqtQLAAAgAElEQVTRtvYDNM9ds2UANVF8xfJm/GZHH0aiKSxv9epjKYzcM3o3SgB44cgIPvvATr3qRAiB5w6P4PKlTfrVzXlaDwUzodEjd5cCvzPfdgHIR2cHBibwh139eObQCN73gxcnbW0YTWZ0gZZRnpFRzZbxG/rmS5KZbIEtMxdf9K1a6V0ontavnIjUyP37fzqqJ3KLV6AKITAYTqLV79AF4fkjIxVVW8idvirZxGU0OnmfgOLIvS+UQCqT0z9TxUixlZaHZNvJcUwkM9h8bAxHtMn9eJkJZyyWxr/8fh9+Y1hYNxZL6602iAitPueMbJnTY3Es1ZLqv9nRp/9dW6pUGnl6PA671YImj33qg88SFvca4EItIn9KK1PzOm24cX0nPvSqpVC3uFVx2KxY2a5Gym+9uAcAcGhQFfV9mt0wHEnmI/eEtGXyE9B1F7TqkduyVi8cNiuavY5Jkbvfpegf0O8+eRg/e/EEnjmkWjHHR2I4PR7HK5Y16Y9p9qqX0mb+vNxRSq2WUccirR+ZNDs4GMHO0yE0e+04NhLFl/9QWG0wEsk/r5n4jEY0cXfm++ZLjJ57NJXVr3ZmEz1yj6f1yL23yYNtJ8fx4+eO4fo17QAmi/toNIVUNod2vxMr2n3oDDjx97/ehQ/c89KUrykjaLOFbdtPjmO/oTRVXj2NxVL6lVhx5C6vHEslI2W5aXFeQFam7B+YwJFh9b2RE48ZsvQzZrB/ZKmspNXnmHZCNZnJYnAiicu0HdW2HB9DT6MLTsWC/5nmCu2ZcmY8gY6gc0pLdzZgca8BGjx2LG/1YttJ9VLW67DhiuXN+KRWYmhkXbeaAL1yeTO6gi4cHIwgFE/r7RJOj8WRE6q/ncmpUaHflU/gXrMi3whtuWYPdQad+uMBVaD8TrUmnQgQArAQ8NAudfPwZ7VWA7J3PQA96jJbWSg3HWnwKHpknY/cNXEfmMCu0yG8+oI2XLeyDc8cKmxnMGR43oODEWw5Poo/7lPHk8xkMZHMoNFjh1OxQLFSwarfRFr13J2K+nUws2aKxXA6RJIZvQonFM9X6VzQ4cOxkRiyOYHPXn8BfE4bhg2T1GN7B/DX/28rAHUdQ8Cl4I+fuhqvW92mfxbKcWw4n6wt9sk/+8BO3H7fNv13Ke6heGnPXV4BGCPzoYkkHth6Sn8dYHKLBCnuBwYm9Mj9xGjpyF1ORsYtHOWVl6TV75i25y4n9A2LgrDb1Pf6siVNuG5lG363s68qtednxuPoLNGyZLZhca8RLultgHQLZN8ZM964vhPXrWzFyg4fzmvz4uBApECUZFQqk7zxdFb33AF18dWaLj9cilW3XToCTvQVRO6qT2+1EDr8Tlx1XjNuWNuBR/cOIJsTeObwMNr9Tv3yF1Dr/v1Omx7dGRmPa5G7y67bMuGEWj8vv8DbTo5jLJbGmq4ANvY24MRorGBxj4x4m70O7OsP43//bBs++tOXEU7kbYZGjx1EBL9TKZiskhnVc1/V4YdiJdOKmb/91U784292lzzv5dh+chw5AXQGnAgn0vrEJa+ybtnUi0VNbrR4HQWT1J1PHMahoQj+8soluEpb9OZUrFjW4sWEyUYuxRi97+KVy4MTCeztC+vVOvlqGYPnXpS4lc/RF4rrNtIDW0/hE7/YjqGJpP644sj9RIG4R7SxxUraX7J1tXFCGo+lCvJLM7Fl5KTT3eDSV3mv7wliXXcAw5FUVaqDzozHq+K3AyzuNcPFixv1n2WvdzMuW9qEu993CRSrBee1enF4KII9WiQacCn6l9nYE6c4L3D7defjY68+T7907AioJXi/2XEG77zreQxH8tH+fR+6HP9xy8V43ep2jERT2HxsFM8dHsErljUVWEYA0OR1mHY1HI+ptfZyY3BAvTqYSGaQSOfQFXQhnVWFYG1XABt71XOx+VjeJ5W2zKaljTgwEMHp8ThiqSzu33JKv0/aSH6XUpCMS6RVzz3gUtDmdxZU0khGoykcLUoGbzk+VlG0J5e6v/L8Fj1yd9utuHpFC16xrAkfvXY5AHViMo5rJJLElcub8fdvWFUwoRs3cokmMyUTzEdHonqTOaPvns0JXYj/sLsP2ZzAeDwNu82CRDqnHxtLFkfu6uvkRF7AQ5p9NhBO5G2ZUByHhyJ4x13PYTyWwsmxOIjUev1oKouV7T4kM6Vr/Ac0uyVWFLk3GGyZFp8DE4lM2ZYWxchKme6gW28PsqEnmC+PnYPFc0bS2RwGwgl0Bee+xh1gca8ZLtH6vltI3ampEtZ2B5HM5HDnE4cRcClY2e7Dce1y2Ji09bsKxf3Vq9rw4Vct03/vCroQTWVxx8MH8NyREew4FdInhJ5GN9x2G65e0QK71YJP/dd2jEZTBZaMpMljx2g0hVxO4D+fOqILgzEqk5NGOJHW96CV3r3NQljR7sPqTj+cigWbj+dX4MpJY5Pmp57X6sWFPUH85LnjGNFWp8rLen/R5Cgjd5fdiga33bT6ZzymWltSTHacGsdbvvss/nSw9IbokiPDUbT7nehucCGRVjt3eh02rOsO4qd/tUkfV7PPXjD5jUQKrQiJT086p3HnE4fwxm8/bTrJHB+J4cJFar7GGLmPRlP6VeAfdvUjFE9DCHVRGgCc0jz14sjd2NpC+u7yKmQgnNBXAZ8ZT+Ch3f14/sgoHt4zgKGJJC5a1KA/9mrN+pO2UTG6LaOd60w2h3BCXZktafVp5ZDT8N1PjauTTHvAiaUtHrjtVqxo9+lefnGSfrYZCCfUK7iFErkTUQ8RPU5Ee4hoNxF9zOQYIqJvEdEhItpBRBfNzXDPXRY1utHsdcDrsE2KiEvxhrUduHlDFwYnkljZ7kOz16E3yWr35z9gU1X0dGiRxpHhqO5VFj/G51TwmetXotXnxPJWL65eMbl3TqPHjpFICrvPhPGl3+3VvdqxWFr/ghk9d/nFvVJrjbyi3QenYoViteDCnmBB5D4cSSLgUrBBE7OPXLMM73tFL44MR/Grl08DAJq89oLXANTGazJydyk2NHjseg5AIj15IG8xSKurkpWOJ0ZiWNTk1iPEM6G4qbXW4nVgWIvcZZ6g2Vta3MOJDM6MqxFzsUUht2HcpG0LaYzc5QRyQYcfW0+M68l2uZmMFP5im6I/HNc7osrzIMti+8OJAs99u5YTkO/xdRfkczny51K+u5yI5EQqI+pCz10uZKrcdz8zHkerzwG7zYKPXrMcv/jQ5VCsFgS0smJpD84V8j1YMOIOIAPgk0KIVQA2AbiNiFYVHXM9gPO0f7cC+O6sjpIBEWHT0kY0ec0blZlhsRC+8tZ1eOeli/D2jT26uAF5wQZQ4LmbIRcyWQj4+tvWA0BBElbygSuX4P6/fgUe/cSr9OoYI01eB0aiST0y3Kn1Lhk3VEJ47TYQqbaMFKzVnQG0+By4eHE++ruktxF7+sJ6cnIkkkKT147VnQE8/qmr8aYLu3DD2g50BJy6uDd61DHJiclqIQRciiFyt6DBrej+s8TY60aWch7T/OyBCiLHYyMx9Da59Unl9Hjc1Fpr9joQTmSQzGR1oTR7v+X4JxJp/ern6HAU4URaT7RK+21VZwA+h63AapLi/vaN3QCg9xRaWrS+wqxaZn1PAHarRU+qysj9yFAU6axAV9CFVCaHZ7WE9/NH1KurTUub4LFb4bZbsaEnCMVKJStm5Fjl5CLfj6DbJHKfhu+urhdQP8tNXgfWaKuv5ed/tnoa7TkTNm2DUc0FTEAF2+wJIfoA9Gk/TxDRXgBdAIwd828C8GOhXhs+T0RBIurQHsvMEp+/cTVC04wuFKsF//LmtQAKo0zjqlkzoTYiFzJddV4LblzfifF4GpcvbSz7GDOavaotI5O6u3RxT+sLtywWgtdhQziR0aOyVr8Dv/rIKwq+3Bf2BJHNCeztC2NjbyOGIkl9QpERqN1G+Ourl+Fz/70bFsp/ieXf67Fb4VSshsjdCptlcqdLoxcrk5TSUhiaInKMJjMYjiSxuMmji3t/KIHu4OSWFM2aYA1HUrqgmdVD+wxrAYzi/sT+Ifzg6aPY/Y+v0yefJc0etAecBdVBUtyvXN4Ml2LFH/cOFpw3QL2iMUbuqUwOw5EUOoMudDW49AoY2Xt/rxb9r+ny4/R4XL/qkNU/ixrdWNHuQzorYLNa0NPgLtn4S77v8aI9AxrNxL3CTeEBVdx7mzyTbpeBhdlK65nwn386gt/u7MPWf3hNwRXaaV3cF6DnTkS9ADYAeKHori4Axr3RTmm3MbNIi8+B5cYNVaeJUSjKJVSLafM5cdOFnbjtGjXxd8umxTMaR6PHjpzIi/qhwQhiqQzGYmk0ePJjkP1lBsNJOBULfA4buhvcBV8UuShKLlgajiTRYhLlvn1jD1p9DgTddn2lsPx7vQ4bHIoVCRm5K6rnPpHIFLQwCBVE7jHt/8oid2lfLGrM2zLprCgZuQPA8ERSF+AmU1smb13JNQJHh6N4+cQYUtkcRiIpfSLvaXShPVCYJB6eUB/TFnBiXXdArxwyVjd1N7gRTeUrcqRV0hFwoqfRrf9dMnKX4i5XSQPAX2xaDEBtp9HkseNf3rwOX3nrOvV8NLlx3CRyl5MIkPfcjaWykga3HTYLmUbuBwYm9PEYGQgn9S0ujcj3JTRLkftYLIVUJjepJ9JgOAmfw1bQO2ouqVjcicgL4H4AtwshZtSAg4huJaLNRLR5aGjh9o2uV4yX+MaE6lS2jMVC+OY7NuDSJdOP1s1ef/sp1TrICWD3mTBC8VRBVC77ywxOJNHqc5rmGLoaXHDYLLq4S1umGKdixZffslafmIC85+512uCwWfQvtctuQ6MnH8V94hfb8P0/HdEjOqdiwfGRKIQQemRcyvMVQiCZyerRaW+Tp2AS9Zl57nrkni8rbPJMnrB8hnLRkNaH5/BgRJ80hyPJAiHpCDgnee52mzppbjAkOpcUiLsLQqjJZiAv7m1+J3oaXJMSqlKApdVhIVXcbRbCokY3iNRkuNygpt1vXsooz6fVQnrkbmxPIbFYCK0+B3718ml849EDBT2LPvffu/DZB3YWPG8inUUonkabyf4LbrsVipVmrVpGfl4e2j1QcPtwJKlfnVWDisSdiBSown6vEOIBk0NOA+gx/N6t3VaAEOIuIcRGIcTGlpbJCTdmbpEJKZtmfTi05KivTGnlbCKvHE6OxvXE5zOHhpHOCr2FMQB9w47BiYR++V2M1UJY1uLFwcEIUpkcQvG0qc8PANeubMMHr1ySf37t7/U4bHAqVj36dSkWfZIZi6Xw0K5+PHlgSL9/TWcAx4ajGJpIIpbKwmqhgsg9nEjrvuqdTxzGtV97Uu+lY0yoytcuRiZPhyPJfPlmmYSq6rmrxz17eETf+nAoksRQJKlPFu0BF4YiSf1qZEi7yiEi/X1wKhY0eux6wlyucZC+u5wcOgIudAZdGI+l1b1diwRxabNahbK81YtmrwNXnteM9YZoPv+3OjASSSJbVOsuz2dPg0uP3OVEZxR3APjsDReg1e/ENx49iO2n8gvPzown9LyORCbnzSJ3IjX3MluRu3yex/cNFmwEMzSRNE2QzxWVVMsQgLsB7BVC3FHisAcBvEermtkEIMR++8JDfrC8TrXixue0wWO3wmatTkWsUaguWtSANr8D3/njIRCpHrpE9nQfnEiW3OkKAJa3enFoMKKXOpoJoRl65K5NcNLfdtmt+gR4fCSGaCqLgXBC/7Ku7wniTCih9+lZ0xXASCSpR42fuX8H/kLb2OTgwAROj8fx0xdOoMGtIOBSCnIbZW2ZSArDUTW6Nquq8WhJ54FwEumsgN1m0YUQUG2doXBe3LuDahSej+xT+mdhg3beG93qAi85ycpFPtJ3l7ZOe8BZMAlFUhnYDEvpGz12bOxtxLUr2wAAd7/3Enz5LWsn/Q0tPgdyYvI2jfIKobfZo0fu4zG1aZjLXlgCfOP6TnzrHRcCUK9cAOgL34YjqQJh7TdceZgRcCmzVgo5Hkuhp9GFSDKjr9YGtMh9GgURZ0sl3+orANwC4Foi2qb9u4GIPkxEH9aO+R2AIwAOAfhPAB+Zm+EyZ4OsFvFonp/XYZtU4z6XGC2G7gYX1nYFkckJ/ONNa/R+H4AaWQ9NJNA3ntB7d5uxvNWL0+NxvXKj0i+OtEc8djVyl7tRuew2PbkmWxD0hxIYj6VhtZDe2/7Rverl9qYljcgJtafKaDSFR/YM4NRoHEIIfUHP6fE4FmlJPIfNqrc4MBNtp2KFz2FTV3tG1C6WZpaUTDrL6HR1p197fvW5hyOpgsj99Wvb0eBWcIfW+nd4Ii8yrX6126SsIW9w26FYSS81jKYyODQ4gV9uOQm/0wa/06a/j8dHYmp9fLNHf3233Yoff+BSfOb6lQDUKyyzv0G+fnFnRzmJ9DZ5kMrmkMnmMBpNm9b7A2puwG614LC28jWcyOjlvsY8w0AF4l5cCtkfSuBbjx2cViO5XE4gFE/jOm1yM+6pMBxJ6e9JNaikWuZpAGULq7Uqmdtma1DM3BB0KbBQ/rLe51TgrOKGyMYVhj0NbnziNU246cJO3Li+s+A4v0vBcCQFu82CN17YWfw0Ostb1QobWUtdvBF5KWQEXRw9u5R85C77qocTGfSFEgi4FFy2tBFehw0/ef44FCvpVxsD4QRePjGuraIVCCcyBT1iFhs2awm4FCTSyZItJFp86irVeDpb9krE71RwSltxeWFPEC+fGMf67iB2nQlpnntC7xPkdyq47Zrl+Kff7sXTB4cxHEnqExUA/PXVy5DTEqcBt4IGt123jQbDSdz2062waXkXItJ946NaPuE87Qqq1GRU6u8EJjdKGwgnYLda9IqSeDo7qWmYEauFsKTZo4u7sXrm9Hhcb0udF3dzcQ267ZPyJ199aD/u33oKN67vLMhHlGMimUFOqJ9Fm4X0K5OprMO5gFeonkNYLIRGj10Xlq6gq2JBnA1sVosu8N2NLqzq9E8SdkCNrmwWwvf+4qKClY3FSHH/+Usnsb47oPeOnwpjtYyMpAHo1TIAsNuwf+j+gTACLgUdARe+8tZ1EEJdmduh+dKD4XzzLEAmRJNYrzVxkys/gXzyupS4L2n2YG9fGCORpGkyVeJz2vRyRDnJrOsOoNnrwIlR1VIyRol/sWkxOgJOfOfxgxiJptDssxfc957LewGoE9HiJjc8mgWyty+MiUQGX3zTGlyzUp0sZO5EtmOQ70PjNPxkae0UR+7HR2LoaXTBpV1dxtPZSU3DilnW6tFzG8YciLHz5+CEanOVKh4IuJSCUsj+UAIPblfThuXq31OZHL7+8H6906e0dhrcdgTd+QVx0jqsprhXJ5PGLBiavQ7divnq29ahOlsU5GnUVoDK/WXN+MCVvbhxfUfZYwD10t1qIWRzAh961bKKo0b593scViTS+ce4tLp3l2ItaCx2cCCiV3ncsLYDf3vDSjgVqx4FPnt4BNtPhXDNihY8vn8IQxNqQvTNF3XjL69aissMawJ0S6iEuF+2tBGP7RuE12HDa1d5TY8BtIoizU5a3urFP9+8Fq9a0YKtJ8b0MkCjuDsVK9516SJ8XbNmSonMF964Gpms0GuyZURs3BZOXlHIiiEp7sUJz3KUityPDkexpNmjt9hIpHIYj6WwqMxWlctavHho9wCSmWxBmwVjA7P+UALtfvPKKwCTEqr3PHtM72dULtH68okxfPuPh7Cy3Y8/W9eRb4LnVgoWxMny02omVFnczzG+dPNa/Yvjm8ZGIrNFk9eh91YphcNmnVLYAbXTZG+TG5mcwOtWt1c8hoBLgdtuRXvAhX7D4h55XhrcCuKhrD5xJDO5Alvg1leqfXfS2RyIgJ+/dAJWC+GDVy7F4/uHcGQoikxOoMljn3RlIiPHUhVKsjdOJJkpa8sY37uAS8G7LlsEQBVtuctRcaXR2zb24N8ePYCcKC3uctKRIiUj4g5Dm1q33Qa33YpjWq3/okY3FCtNawMKedVkjNxzObXE9JXnN+ub0MTSmUlNw4pZ3upFNidwYiSmNyPzOmwFE7S6D23pqDngUpP42ZwAAbjvpRM4v82LAwORsuIuK3nkJGXcMrLBbddtmaGIOpYFVwrJ1A8XL27Aqs7K7Iu54KJFDbjcsInH2fL1t1+I/7jl4oq3MgTUKPbhj78Sf76xBw5bvgJDVmPI5KLR5gmaXM4r2o46sVQWrzq/BSs71IVV+/vVyNlMQP1T2DKrOvz6feVaTRgnB+MaAaN4FCfv2gNO3Yefyh6QC20OD0VgtdCk52ry2vWFTAGXgrdv7MF1F7SVfU4jRIRmr6MgN9EXTiCZyWFJs1efaNVN3AubhhUjVzcfGoyo9f1OG5Y0ewoid7XyqnRyXk7e4Xgae/rCGIul8ZaL1PYMxg3bixkpFndtIgi47Ai681aPjNzNFtrNFSzuTFX5zPUrcee7L56157uwJ6j3RZ8O3Q1u2G2WSZ47kLcXlrd69cVGwRKWQ4tWzfPmi7rQ4LbDQvktDc0i74CrvC1js1r0DqDlImEp7lYL6f44UCjaZmsEPnDlEvgcNn0bxlLIyHk8lkarzzFp8mzyOJDRqkh8TgVfunmtaf6kHC2+whbQ0sPvbXbrE62sry/nuctk5+GhiBahO7WFW6q4CyHU28tUXhnb/j6vbWcprwanE7mHYkZbxhi5J/W/uVqwuDPnNOUi986gU1/JWyoR1+53wOe04dUXtMGqJazlrktmCVH/FLYMAL0stBJbJuhSCnzkFu0xVguZeuBXLG/Gji+8tmwUCxS2lS7e0xco9I5nugiuuH+9rL5ZaojcpfdfanIF1ImyM+DEocGIbr90Bl36RuITyQxiqSzaA6WFVW/7G0/jucMjWNLsweImtcyyEnEfmpCbncjIXUHQo0buQggMR5KaFVVZu+7ZgMWdOacxj9zVL3pn0DWluH/qdSvwH39xsf6lbfY69C+4WfJsWYsHQW1RUymuX9OONV1+rO4MlDxGCmrx88jIvdlrL7lPZyWJZ4uF9OjdbFs4OXGpVz8zEyyzyN2lJarlRCutlcYpkrUbFjXg6UPD6AupEXpn0IlIUt0A/viw3KCmdGWYbPs7EknihaOj2LRU3WzG71IQjmdKPm6SLRNLw+uwQbFa0OC2I5XNIZbKFiwcqxacUGXOaaQwWS0ExaqKnox4OwMufdFLqTrrYgFWxVWN3M184jeu78Tr17QXXDEUs7jJg9/8r6vKjltG7sWL0ORlf7nFX5Xittu0iHfyc+m98c+idUWz14GRaAqZbA42qwVHhyPobfaAiPSJVoq7sWmYGTeu78Bvd6qL4lu0yF0+/pnD6oYql5XpjSQnyWcOjSCSzOh5oYDLVrYUUm5QkvfcU/pzySBhLJYqWDhWLThyZ85p5KpOt2LVI9qCyH0KcS9GRmcBlwLFpK0DEZUV9kqRolo8Likgs+HtehzqODtMxb2wN/5MaPE5IAT0na+OjcSwpFmtkpKR+2nNWpmqzPLqFa16fqTN59Srrfb3T+Dpg8NY0earKKH6yy0nYbUQLtesMdnnqBSyB9BwJAkhBEKGjWf0PkXRdMGK4WrB4s6c0zi0CNFpSEped0Eb3n9FL5a1eAy2TGWX1FJcK+1zM1NK2jJ65H72QiIrZsp57mfTdK7FsJApnc3hxGhMT45OitynEHenYsVrtQRom9+JtV0B9DS68INnjuHFY6P6bl6lkOdxIpHBF964WhfiqRqKSc89kc4hmspiPJ4X9wZDE7pq95UBWNyZcxwZuRsTiD2Nbnz+xtVa5Uojzm/zTlldIpHi2lxmdelsYEyoGvFoHRnXdpf26ytFVuGYRu7a33c2ayXyC5lSODMeRzYnsLixUNxD8bRp0zAz3npxNyykVjlZLYT3Xt6L7SfHkcrkphR3xWrBqy9oxT+8YRVu0frQA+qVSSlxF0JgLJbS6+eHJ5LqrmJaICDbRw+E1f5E1RZ39tyZcxrpubtLiMeKdh8e/virKn6++Y7ciQiPfqLy8ZbDrdkcZonIplmI3GVeYCCUgKIlf2U7DIuF4LBZkMzkyi5gMnL5sia8/LnX6ufkbRt7cMcjB5DJirJ+u+T7771k0m0BlzLJcz8zHsfH79uGL928FumswPltPgyE1Q1WQvE0AkW2zAtH1a0Gz2srveJ4LmBxZ85pZOQ+WyVq0q6Ya3H3l0ioziYeuxUWMvfv8wnVmb9+e8AJq4XUjT+0Ap4uQ68jl92qivs0Vr4aJ7uAS8HHX30+BsKJGe9+FHApCCfUHalkTuapA0N44ego/nub2ntmRZsPf9Iaso3H0vrVlPz/6YNqQndNmeqnuYDFnTmnkaLumjVxV4WwcY5tmY6AE59+/Uq8Yd30Fg5NhyavHd0NbtPEsNr//ewid8VqQVfQheMjMViIQFTU5kCxYhyl2/1Wwl+9cumMHwuoHUSzOYFIMqNbUHu03j1PHVB3kzu/XV2ZfGwkhkxO6J67zWqBz2lDfzgBn9OGnsbqNekDWNyZcxzdc6/A062EDi0a7Z7jHe6J1M2/55JPvmYF/uoqc3G0WS348pvX4uLFpbt2VsIibT9Wu82CVp9D3wkKyCe5p9OQbLaRVwLhhEHctXbQO7TOoctbvSBSK3MA6J47AH1P3jWdgYob280WLO7MOY0euc+SuDd5HXjwo1foG3jXMg0ee1lL5M8vWXTWr9HT6MbDu/vhUqwFnSeByYvK5gNpO4ViaXQFXcjlhN51U2uBj1afAw1uO367ow9EwMW9+Qmvwa3gxCiwpqv6/Zy4WoY5pzGrljlbVncGCiJQpjSLGt0YiaZwcHACXUWdQN1F7SDmg3zkriZVZa984xVLk8eBZq+6GvV1q9r1RmZAPqm6pqu6fjvA4s6c48y2585MD9mnfTiSmhS5y/fmbDz3s0UmrGU5pPTb33ax2jFSlmnKXEuxVSavOsq1kpgrWNyZcxqbhWCzEFq+NkYAAAeASURBVNwOFvf5wLgJR1eDuS1TrmnYXBMoFvczYVgthD9b1wHFSvrEc90FbXjnpT1Yb9joHVD/pqBbqXibvtmEPXfmnIaI8LW3rde3qmOqi1Hci5PQ0paZqmnYXCIj93A8jV2nQ3hs3yCWt3jhcypY0e6D1aLGxx+8conp4z9y9XK889JF09pvYLZgcWfOed60oWu+h3DOEnAr8GtbBk6K3HXPff4Sqj6HDUTAT188gX/67V7YbRb8wxtWAQC+eNMa5KbYp9LjsJXs3T/XsLgzDDOvLG7yYOfpUEnPfT5LIS0Wgs9hw5GhKK46rxnffucG3SbaUGbz9oXAlJ47Ef2AiAaJaFeJ+wNE9D9EtJ2IdhPR+2d/mAzD1CuLGt0IupVJEa57AdS5A2p5a7vfiW++Y8O8+v/TpZLI/R4A3wHw4xL33wZgjxDiRiJqAbCfiO4VQqRKHM8wDKPz0WuX42YTa+yN67vgdyqztgZhptzx9vVocNvntWpnJkwp7kKIp4iot9whAHykLr/yAhgFUHrrEoZhGAMXdPhxQcfkRT4r2n1Y0T7/i8EWuv1SitkohfwOgAsAnAGwE8DHhBA5swOJ6FYi2kxEm4eGhmbhpRmGYRgzZkPcXwdgG4BOABcC+A4Rma61FULcJYTYKITY2NLSMgsvzTAMw5gxG+L+fgAPCJVDAI4CWDkLz8swDMPMkNkQ9xMArgMAImoDsALAkVl4XoZhGGaGTJlQJaKfAbgaQDMRnQLweQAKAAghvgfgiwDuIaKdUFvuf1oIMTxnI2YYhmGmpJJqmXdOcf8ZAK+dtRExDMMwZw03DmMYhqlDWNwZhmHqEBJiis43c/XCREMAjs/w4c0AFqqvv1DHxuOaHgt1XMDCHRuPa3rMdFyLhRBT1pLPm7ifDUS0WQixcb7HYcZCHRuPa3os1HEBC3dsPK7pMdfjYluGYRimDmFxZxiGqUNqVdzvmu8BlGGhjo3HNT0W6riAhTs2Htf0mNNx1aTnzjAMw5SnViN3hmEYpgw1J+5E9Hoi2k9Eh4joM/M4jh4iepyI9mg7UH1Mu/0LRHSaiLZp/26Yh7EdI6Kd2utv1m5rJKJHiOig9n/Vm1QT0QrDedlGRGEiun0+zpnZDmOlzhGpfEv7zO0goouqPK6vEtE+7bV/RURB7fZeIoobztv3qjyuku8bEX1WO1/7ieh1czWuMmO7zzCuY0S0Tbu9mueslEZU53MmhKiZfwCsAA4DWArADmA7gFXzNJYOABdpP/sAHACwCsAXAHxqns/TMQDNRbd9BcBntJ8/A+BfF8B72Q9g8XycMwCvBHARgF1TnSMANwD4PdTeSZsAvFDlcb0WgE37+V8N4+o1HjcP58v0fdO+B9sBOAAs0b6z1mqOrej+rwP43Dycs1IaUZXPWa1F7pcCOCSEOCLUbfx+DuCm+RiIEKJPCLFV+3kCwF4Ak/cKWzjcBOBH2s8/AvCmeRwLoHYSPSyEmOlCtrNCCPEU1F3DjJQ6RzcB+LFQeR5AkIg6qjUuIcTDQgi5u9nzALrn4rWnO64y3ATg50KIpBDiKIBDUL+7VR+btkPc2wH8bK5evxRlNKIqn7NaE/cuACcNv5/CAhBUUrch3ADgBe2mj2qXVT+YD/sD6taHDxPRFiK6VbutTQjRp/3cD6BtHsZl5B0o/MLN9zkDSp+jhfS5+wDU6E6yhIheJqInieiqeRiP2fu2kM7XVQAGhBAHDbdV/ZwVaURVPme1Ju4LDiLyArgfwO1CiDCA7wJYBnVXqj6ol4TV5kohxEUArgdwGxG90ninUK8B561MiojsAN4I4JfaTQvhnBUw3+fIDCL6O6j7E9+r3dQHYJEQYgOATwD4KZXYBW2OWHDvmwnvRGEQUfVzZqIROnP5Oas1cT8NoMfwe7d227xARArUN+1eIcQDACCEGBBCZIW6j+x/Yg4vR0shhDit/T8I4FfaGAbkJZ72/2C1x2XgegBbhRADwMI4ZxqlztG8f+6I6H0A3gDg3ZogQLM9RrSft0D1ts+v1pjKvG/zfr4AgIhsAN4M4D55W7XPmZlGoEqfs1oT95cAnEdES7To7x0AHpyPgWhe3t0A9goh7jDcbvTIbgawq/ixczwuDxH55M9Qk3G7oJ6n92qHvRfAf1dzXEUURFPzfc4MlDpHDwJ4j1bNsAlAyHBZPecQ0esB/B8AbxRCxAy3txCRVft5KYDzUMVd0Mq8bw8CeAcROYhoiTauF6s1LgOvBrBPCHFK3lDNc1ZKI1Ctz1k1ssaz+Q9qRvkA1Bn37+ZxHFdCvZzaAXWD8G3a2H4CYKd2+4MAOqo8rqVQKxW2A9gtzxGAJgCPATgI4FEAjfN03jwARgAEDLdV/ZxBnVz6AKShepsfLHWOoFYv/Lv2mdsJYGOVx3UIqhcrP2ff0459i/YebwOwFcCNVR5XyfcNwN9p52s/gOur/V5qt98D4MNFx1bznJXSiKp8zniFKsMwTB1Sa7YMwzAMUwEs7gzDMHUIizvDMEwdwuLOMAxTh7C4MwzD1CEs7gzDMHUIizvDMEwdwuLOMAxTh/x/WxBRsaMKLHAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "%matplotlib inline\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(all_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating at different \"temperatures\"\n",
    "\n",
    "In the `evaluate` function above, every time a prediction is made the outputs are divided by the \"temperature\" argument passed. Using a higher number makes all actions more equally likely, and thus gives us \"more random\" outputs. Using a lower value (less than 1) makes high probabilities contribute more. As we turn the temperature towards zero we are choosing only the most likely outputs.\n",
    "\n",
    "We can see the effects of this by adjusting the `temperature` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "th you good plath, t art the thour oft thno graiie, in and the thour muth my hearveth; with qure and of shade. ear.  merath; ponce in the hand am art pont; me, and to me ome so me comes grear fome would\n"
     ]
    }
   ],
   "source": [
    "print(evaluate('th', 200, temperature=0.8))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lower temperatures are less varied, choosing only the more probable outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ther fare the pother for the preath are to me to me to maine the preath him. i manders.  pring me some me to me of to me to me to me to me as the preath.  preart: are the preath are the preare and the h\n"
     ]
    }
   ],
   "source": [
    "print(evaluate('th', 200, temperature=0.2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Higher temperatures more varied, choosing less probable outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "thut. grie vank, you he asrets some quang! su, ixeseer? 'mynust! to maon pimig--horque a! ee, uto 'kinghild obry o'd arwelf gahesed! eard.? frth uncar: coldef viuth'd pom proead thom pry. liodintw loant\n"
     ]
    }
   ],
   "source": [
    "print(evaluate('th', 200, temperature=1.4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise\n",
    "\n",
    "Modify the code to use batches of sentences and remove the for loop in the  function `train`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why build a language model?\n",
    "\n",
    "Building a language model is not very useful by it-self. However language models have recently been shown to be useful for **transfer learning** to build **contextualized word embeddings** as a better alternative to word2vec or GloVe.\n",
    "\n",
    "Using language-model based word representations makes it possible to reach the **state-of-the-art at many natural language understanding problems**.\n",
    "\n",
    "The workflow is the following:\n",
    "\n",
    "- **train** a (bi-directional) **deep language model on a very large, unlabeled corpus** (e.g. 1 billion words or more);\n",
    "- plug the resulting **language model as the input layer** (and sometimes also the output layer) **of a task specific architecture**, for instance: text classification, semantic role labeling for knowledge extraction, logical entailment, question answering and reading comprehension;\n",
    "- **train the task specific parameters** of the new architecture on the **smaller task-labeled corpus**;\n",
    "- optionally fine-tune the full architecture on the task-labeled corpus if it's big enough not to overfit.\n",
    "\n",
    "More information on this approach:\n",
    "\n",
    "- Deep contextualized word representations, https://arxiv.org/abs/1802.05365\n",
    "- [Pytorch implementation of ELMo](https://github.com/allenai/allennlp/blob/master/allennlp/modules/elmo.py) as part of the AllenNLP project: https://github.com/allenai/allennlp\n",
    "- Universal Language Model Fine-tuning for Text Classification: https://arxiv.org/abs/1801.06146 (ULMFiT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
